Extracting Support Data for a Given Task

We report a novel possibility for extracting a small subset of a data base which contains all the information necessary to solve a given classification task: using the Support Vector Algo rithm to train three different types of handwritten digit classifiers, we observed that these types of classifiers construct their decision surface from strongly overlapping small (k: 4%) subsets of the data base. This finding opens up the possibiiity of compressing data bases significantly by disposing of the data which is not important for the solution of a given task. In addition, we show that the theory allows us to predict the classifier that will have the best generalization ability, based solely on performance on the training set and characteristics of the learning machines. This finding is important for cases where the amount of available data is limited. Introduction Learning can be viewed as inferring regularities from a set of training examples. Much research has been devoted to the study of various learning algorithms which allow the extraction of these underlying regularities. No matter how different the outward appearance of these algorithms is, they all must rely on intrinsic regularities of the data. If the learning has been successful, these intrinsic regularities will be captured in the values of some parameters of a learning machine; for a polynomial classifier, these parameters will be the coefficients of a polynomial, for a neural net they will be the weights and biases, and for a radial basis function classifier they will be weights and centers. This variety of different representations of the intrinsic regularities, however, conceals the fact that they all stem I?--^ --------A M”lll a C”,ll‘ll”ll T”“b. In the present study, we explore the Support Vector Algorithm, an algorithm which gives rise to a number *permanent address: Max-Planck-Institut fiir biologische Kybernetik, Spemannstrafle 38, 72076 Tiibingen, Germany ‘supported by ARPA under ONR contract number N00014-94-G-0186 252 KDD-95 of different types of pattern classifiers. We show that the algorithm allows us to construct different classifiers (polynomial classifiers, radial basis function classifiers, and neural networks) exhibiting similar performance and relying on almost identical subsets of the training set, their support vector seZs. In this sense, the support vector set is a stable characteristic of the data. In the csse where the available training data is limited, it is important to have a means for achieving the best possible generalization by controlling characteristics of the learning machine. We use a bound of statistical learning theory (Vapnik, 1995) to predict the degree which yields the best generalization for polynomial classifiers. In the next Section, we follow Vapnik (1995), Baser, Guyon & Vapnik (1992), and Cortes & Vapnik (1995) in briefly recapitulating this algorithm and the idea of Structural Risk Minimization that it is based on. Following that, we will present experimental results obtained with support vector machines. The Support Vector Machine Structural Risk Minimization For the case of two-class pattern recognition, the task of learning from examples can be formulated in the following way: given a set of functions {ja : a E A}, ja : RN + (-l,+l} (the index set A not necessarily being a subset of R”) and a set of examples (x1,Yl),...,(w,w)~ xi E RN, w E (-1, +l}, each one generated from an unknown probability distribution Hr. ul. we want to find a function f.4 which -. ----~ ~---,” provides th&&ll&t p&ible value for the risk R(a) = J KY(x) YI dP(x, Y). The problem is that R(a) is unknown, since P(x, y) is unknown. Therefore an induction principle for risk minimization is necessary. From: KDD-95 Proceedings. Copyright © 1995, AAAI (www.aaai.org). All rights reserved.